Organizing present and future reads from a tiered streaming data storage layer

ABSTRACT

Implementations are provided herein for organizing present and future reads from a tiered streaming data storage layer. Implementations allow for access to multi-tiered streaming data organized in different append-only segments, some of which may be related to each other. Streaming data can be read from fast local tier 1 storage, streaming data can be retrieved from fold tier 2 storage, and registrations can be made to read streaming data that has not yet been written to the storage layer.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/503,845 filed May 9, 2017.

FIELD OF THE INVENTION

This invention relates generally to processing data, and moreparticularly to processing streaming data.

BACKGROUND OF THE INVENTION

Streaming applications are applications that deal with a large amount ofdata arriving continuously. In processing streaming application data,the data can arrive late, arrive out of order, and the processing canundergo failure conditions. It can be appreciated that tools designedfor previous generations of big data applications may not be ideallysuited to process and store streaming application data.

Enabling streaming applications to store large amounts of data from astorage perspective can be challenging. There is a need to determine theproper storage primitive that would ideally be suited for building a newgeneration of streaming applications in conjunction with existing toolslike Apache Flink. In using a Lambda architecture, a developer may use acomplex combination of middleware tools that include batch stylemiddleware influenced by platforms like Apache Hadoop and continuousprocessing tools like Apache Storm, Apache Samza, Apache Kafka andothers. Batch style processing can be used to deliver accurate butpotentially out of data analysis of data. It can be appreciated that“real-time” processing may deliver faster results but could come at acost of reduced accuracy. Furthermore, there may be a need for twocopies of application logic because the programming models of a speedlayer are different than those used in a batch layer.

Conventionally, Lambda architectures may be expensive to develop andexpensive to deploy and manage in production. In some implementations,as more application like Internet of Things (“IoT”), require continuousprocessing, it may not be beneficial to use Lambda architectures andconventional style middleware. Therefore, there exists a need for asimpler approach to Lambda to process streaming application data.

SUMMARY

The following presents a simplified summary of the specification inorder to provide a basic understanding of some aspects of thespecification. This summary is not an extensive overview of thespecification. It is intended to neither identify key or criticalelements of the specification nor delineate the scope of any particularimplementations of the specification, or any scope of the claims. Itssole purpose is to present some concepts of the specification in asimplified form as a prelude to the more detailed description that ispresented in this disclosure.

In accordance with an aspect, a lookup request can be received from acaller, wherein the lookup request includes a segment, an offset, and amaximum length. An iterator can be established based on the segment andthe offset that returns for each iteration until the maximum length isreached: a next set of data responsive to the lookup request wherein thenext set of data is stored in local cache memory; a storage read result;or a future read result.

In accordance with another aspect, a notification can be received formthe caller for a future read based on the future read result. The futureread can be registered in a read index. In response to the read indexindicating that a segment associated with the future read has a newappend, a set of data responsive to the future read can be determined.The set of data can be returned to the caller.

The following description and the drawings set forth certainillustrative aspects of the specification. These aspects are indicative,however, of but a few of the various ways in which the principles of thespecification may be employed. Other advantages and novel features ofthe specification will become apparent from the detailed description ofthe specification when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example illustration of a Lambda architecture forprocessing streaming application data;

FIG. 2 illustrates an example illustration of a Pravega architecture forprocessing streaming application data in accordance with implementationsof this disclosure;

FIG. 3 illustrates an example of multiple individual applications orjobs in a streaming application data pipeline in accordance withimplementations of this disclosure;

FIG. 4 illustrates an example data processing environment for processingstreaming application data in a conventional system;

FIG. 5 illustrates an example data processing environment for processingstreaming application data in accordance with implementations of thisdisclosure;

FIG. 6 illustrates an example Pravega Cluster in accordance withimplementations of this disclosure;

FIG. 7 illustrates an example Pravega Cluster ingesting data from a setof writers and making data available to a set of reader groups inaccordance with implementations of this disclosure;

FIG. 8 illustrates an example Stream decomposed into a set of StreamSegments in accordance with implementations of this disclosure;

FIG. 9 illustrates an example graph depicting the relationship betweenrouting keys, stream segments, and time in accordance withimplementations of this disclosure;

FIG. 10 illustrates an example graph depicting Events and theirrelationship between routing keys, stream segments, and time inaccordance with implementations of this disclosure;

FIG. 11 illustrates an example of Stream Segments and associatedReaderGroups in accordance with implementations of this disclosure;

FIG. 12 illustrates an example graph depicting Events published into aTransaction and their relationship between routing keys, streamsegments, and time in accordance with implementations of thisdisclosure;

FIG. 13 illustrates an example state synchronizer in accordance withimplementations of this disclosure;

FIG. 14 illustrates an example implementation of a set of streamingclients, a Pravega Cluster, the autoscaling of segments, and tieredstorage in accordance with implementations of this disclosure;

FIG. 15 illustrates an example append-only log in accordance withimplementations of this disclosure;

FIG. 16 illustrates an example flow diagram method for organizingpresent and future reads from a tiered streaming data storage layer inaccordance with implementations of this disclosure;

FIG. 17 illustrates an example flow diagram method for organizingpresent and future reads from a tiered streaming data storage layerincluding processing storage read results in accordance withimplementations of this disclosure; and

FIG. 18 illustrates an example flow diagram method for organizingpresent and future reads from a tiered streaming data storage layerincluding processing future read results in accordance withimplementations of this disclosure.

DETAILED DESCRIPTION

The innovation is now described with reference to the drawings, whereinlike reference numerals are used to refer to like elements throughout.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of this innovation. It may be evident, however, that theinnovation can be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing the innovation.

As used herein, the term “node” refers to a physical computing device,including, but not limited to, network devices, servers, processors,cloud architectures, or the like. In at least one of the variousimplementations, nodes may be arranged in a cluster interconnected by ahigh-bandwidth, low latency network backplane. In at least one of thevarious implementations, non-resident clients may communicate to thenodes in a cluster through high-latency, relatively low-bandwidth frontside network connections, such as Ethernet, or the like.

As used herein, the terms “Stream” or “Pravega” refers to a new storageprimitive. A stream is ideally suited to the continuous processing ofunbounded data. In many implementations, a stream may be a name,durable, append-only and unbounded sequence of bytes. In mostimplementations, with this primitive, and with the key featuresdiscussed in this disclosure, the stream can be combined with streamprocessing engines such as Apache Flink to build streaming applications.

Implementations are provided herein for organizing present and futurereads from a tiered streaming data storage layer. Implementations allowfor access to multi-tiered streaming data organized in differentappend-only segments, some of which may be related to each other.Streaming data can be read from fast local tier 1 storage, streamingdata can be retrieved from fold tier 2 storage, and registrations can bemade to read streaming data that has not yet been written to the storagelayer.

In one implementation, an index is created for each segment. Actualstreaming data can be stored in a shared key-value database where thekey is the segment name+an offset and the value is byte data. It can beappreciated that this can allow for inserting data, deleting data andreading data from the cache. The cache is shared between all segments intier 1 storage.

In one implementation, an in-memory ordered index can be implemented asa binary search tree. The index key is the starting offset for the blockof data that the index contains. The value is a redirection to thecache, rather than the index storing the data, and a cache entrygeneration number that can be used to remove cold data from the cache tomake room for new data.

In one implementation, a lookup request can be received from a callerwherein the lookup request includes a targeted segment, an offset withthat segment, and a maximum length of the lookup request. An iteratorcan then be established that returns something with each iteration. Ifthe current offset exists in the index and the data is stored withinTier 1 storage, the data is returned to the caller right away. If thecurrent offset does not exist within the index, a storage read resultcan be sent to the caller. If the current offset is located beyond theend of an unsealed segment, a future read result can be sent to thecaller.

In one implementation, a storage read result asks the caller whetherthey want the data to be loaded into tier 1 storage from tier 2 storageto serve lookup requests by the caller. It can be appreciated that ifthe caller agrees to the storage read result, more data in the segmentbeyond the read can be loaded into tier 1 storage from tier 2 forread-ahead purposes guessing that it's likely the caller will want thatdata in future iterations or future lookup requests.

In one implementation, a future read result asks the caller whether theywant to register their lookup in a read index that when data iseventually written into the system, can serve the lookup request.

In one implementation, the read index can support merging one segmentinto another with a pure metadata change. The segment merger can createa special index entry at the merge offset in the target segment with thelength of the source segment. This special index entry can have apointer to the source's segment index. It can be appreciated that anyreads within this range will be redirected to the source's segment readindex with a proper offset adjustment. It can be further appreciatedthat this can allow two original segment files to still exist in Tier 2storage, but appear as one segment for upstream callers. It can also beappreciated that when the source segment has been fully merged in Tier2, the index entry can either be deleted and future requests to thatrange would result in storage read results or, for added efficiency, itmay be fully replaced by the source segment's read index entries, whichare now part of the target segment's read index.

Referring now to FIG. 1, there is illustrated an example illustration ofa Lambda architecture for processing streaming application data. In someprior implementations, in using a Lambda architecture, a developer mayuse a complex combination of middleware tools that include batch stylemiddleware influenced by platforms like Apache Hadoop and continuousprocessing tools like Apache Storm, Apache Samza, Apache Kafka andothers. Batch style processing can be used to deliver accurate butpotentially out of data analysis of data. It can be appreciated that“real-time” processing may deliver faster results but could come at acost of reduced accuracy. Furthermore, there may be a need for twocopies of application logic because the programming models of a speedlayer are different than those used in a batch layer. With duplicatedstorage and middleware in both the speed layer and the batch layer, itcan be difficult to determine the source that has the actual source ofcorrect real time data. Merging the logic of the speed layer and thebatch layer in the serving layer that interfaces with the applicationclient can also be challenging. Thus, conventionally, Lambdaarchitectures may be expensive to develop and expensive to deploy andmanage in production.

FIG. 2 illustrates an example illustration of a Pravega architecture forprocessing streaming application data in accordance with implementationsof this disclosure. In many implementations, there can be one set ofmiddleware to deploy and operate. In some implementations, there can beone expression of application logic to develop and maintain rather thantwo. There can be one ingest process from the source data into Pravega.One source of truth for data can lie within Pravega. There can be oneembodiment of application. There can be one set of middleware. It can beappreciated that this can create a more stable and a simpler to developfor application environment for the Application Client.

FIG. 3 illustrates an example of multiple individual applications orjobs in a streaming application data pipeline in accordance withimplementations of this disclosure. It can be appreciated that in mostimplementations, most applications, including Internet of Things (“IoT”)applications, involve a pipeline of several, often many more individualapplications or jobs. IN some implementations, a Kappa environment mayhave requirements imposed on middleware in terms of chains of pipelinesof data processing. For example, some requirements may involve end-endsemantics in addition to component by component capabilities. In someimplementations, the current disclosure considers scaling in terms ofend-end scaling, in addition to scaling a single component. In certainimplementations, the current disclosure considers ordering, exactly oncedelivery semantics, fault tolerance, etc., across the entire pipeline.In some implementations, the current disclosure can enable coordinationof the output of one component with the input of a downstream componentto deliver a pipeline of stream components more efficiently.

Exactly Once Semantics

In some implementations, exactly once semantics can mean the currentdisclosure can ensure that data is delivered and processed exactly once,with exact ordering guarantees, despite failures in clients, servers, orthe network. In certain implementations, to achieve exactly oncesemantics, Pravega Streams may be durable, ordered, consistent andtransactional.

In most implementations, ordering can mean that data is seen by readersin the order it is written. In some implementations of the currentdisclosure, data may be written along with an application-definedrouting key. In certain implementations, the current disclosure may makeordering guarantees in terms of routing keys. In many implementations,two pieces of data with the same routing key may be read by a reader inthe order they were written. In most implementations, the currentdisclosures may enable ordering guarantees allow data reads to bereplayed (e.g. when applications crash) and the results of replaying thereads may be the same.

In most implementations, consistency may mean that Readers see the sameordered view of data for a given routing key, even in the face offailure. In most implementations, systems that are “mostly consistent”may not sufficient for building accurate data processing.

Conventionally, systems that provide “at least once” semantics maysuffer from the possibility of data duplication. In typical systems, incertain failure scenarios, a data producer, upon recovery may write thesame data twice. In most implementations of the current disclosure,writes may be idempotent, rewrites done as a result of failure recoverymay not result in data duplication. In most implementations, it may notbe possible to deliver exactly once processing semantics when there isthe possibility of data duplication.

In certain implementations, techniques may extend beyond just exactlyonce semantics between two components within an applicationarchitecture, and may include end-end exactly once semantics across anentire pipeline of streaming application components with strict orderingof data in the face of component failures. In some implementations, datastorage may be the foundation for once across an entire data processingpipeline. In many implementations, data storage may be important forbuilding streaming applications that are both timely and accurate. Inmost implementations, without accuracy at the storage layer, applicationdevelopers may be faced with relying on the complexity of a Lambdaarchitecture to deliver accurate data analysis in combination with nearreal-time processing of the data.

Auto Scaling

Unlike systems with static partitioning, the current disclosure mayautomatically scale individual data streams to accommodate changes indata ingestion rate.

In a particular implementation, imagine an IoT application with millionsof devices feeding thousands of data streams with information aboutthose devices. In this implementation, imagine a pipeline of Flink jobsthat process those streams to derive business value from all that rawIoT data: predicting device failures, optimizing service deliverythrough those devices, or tailoring a customer's experience wheninteracting with those devices. In this implementation, building such anapplication at scale is difficult without having the components be ableto scale automatically as the rate of data increases and decreases.

Conversely, with implementations of the current disclosure, it may beeasy to elastically and independently scale data ingestion, storage andprocessing—orchestrating the scaling of every component in a datapipeline. In many implementations, the current disclosure may supportauto-scaling starts with the idea that Streams are partitioned intoStream Segments. In most implementations, a Stream may have 1 or moreStream Segments, any data written into the Stream may be written to oneof the Stream Segments based on a routing key. In certainimplementations, writers may use application-meaningful routing keyslike customer-id, timestamp, machine-id, so like data is groupedtogether.

In some implementations, a Stream Segment may be a fundamental unit ofparallelism in Pravega Streams. In many implementations, a Stream withmultiple Stream Segments may support more parallelism of data writes;multiple Writers writing data into the different Stream Segmentspotentially involving all the Pravega Servers in the cluster. In certainimplementations, on the Reader side, the number of Stream Segments mayrepresent the maximum degree of read parallelism possible. In certainimplementations, if a Stream has N Stream Segments, then a ReaderGroupwith N Readers may consume from the Stream in parallel. In mostimplementations, increasing the number of Stream Segments may increase anumber of Readers in a ReaderGroup to increase the scale of processingthe data from that Stream. In further implementations, as the number ofStream Segments decreases, the number of Readers may be reduced.

In some implementations, a Stream may be configured to grow the numberof Stream Segments as more data is written to the Stream, and to shrinkwhen data volume drops off. In certain implementations, growing andshrinking a stream may be referred to herein as the Stream's ServiceLevel Objective or SLO. In many implementations, the current disclosuremay enable monitoring a rate of data input to the Stream and uses theSLO to add or remove Stream Segments from a Stream. In certainimplementations, segments may be added by splitting a Segment. Infurther implementations, segments may be removed by merging twoSegments. In most implementations, the number of Stream Segments mayvary over time.

In certain implementations, it may be possible to coordinate the autoscaling of Streams in Pravega with application scale out. In furtherimplementations, using metadata available from Pravega, applications mayconfigure the scaling of their application components. In a particularimplementation, scaling may drive the number of instances of a Flinkjob. In another implementation, it may be possible to use software suchas Cloud Foundry, Mesos/Marathon, Kubernetes or the Docker stack todeploy new instances of an application to react to increased parallelismat the Pravega level, or to terminate instances as Pravega scales downin response to reduced rate of data ingestion.

Distributed Computing Primitive

In most implementations, the current disclosure may enable efficient useof distributed applications, such as micro-services. In certainimplementations, the current disclosure may enable data as a datastorage mechanism, for messaging between micro-services and for otherdistributed computing services such as leader election.

In some implementations, a State Synchronizer may be a basis of sharingstate across a cluster with consistency and optimistic concurrency. Infurther implementations, a State Synchronizer may be based on afundamental conditional write operation, so that data may be written ifit would appear at a given position in the Stream. In certainimplementations, if a conditional write operation cannot meet thecondition, it may fail. In certain implementations, a State Synchronizermay be a strong synchronization primitive that may be used for sharedstate in a cluster, membership management, leader election and otherdistributed computing scenarios.

Write Efficiency

In many implementations, the current disclosure may enable shrinking ofwrite latency to milliseconds, and may seamlessly scale to handle highthroughput reads and writes from thousands of concurrent clients, makingit ideal for IoT and other time sensitive applications.

In implementations of the current disclosure, streams may be lightweight, and implementations of the current disclosure may be able tosupport millions of Streams. In many implementations, the techniques ofthe current disclosure may free an application from worrying aboutstatically configuring streams and pre allocating a small fixed numberof streams and husbanding or limiting stream resource.

In many implementations, the current disclosure may enable low latencyWrite operations, for example, under 10 ms to return an acknowledgementto a Writer. In further implementations, writes may be optimized so thatI/O throughput may be limited by network bandwidth rather than apersistence mechanism being the bottle neck. In certain implementations,Apache BookKeeper may be used to persist all write operations. In someimplementations, BookKeeper may persist and protect the data veryefficiently. In certain implementations, as data may be protected beforethe write operation is acknowledged to the Writer, data may be durable.In many implementations, data durability may be a fundamentalcharacteristic of a storage primitive. In many implementations, to addfurther efficiency, writes to BookKeeper may often involve data frommultiple Stream Segments, so the cost of persisting data to disk may beamortized over several write operations. In most implementations of thecurrent disclosure, there may be no durability-performance trade-off aswrites may be both durable and high performance.

In most implementations, reads may be efficient. In certainimplementations, a reader may read from a Stream either at the tail ofthe Stream or at any part of the Stream's history. Unlike some log-basedsystems that use the same kind of storage for tail reads and writes aswell as reads to historical data, implementations of the currentdisclosure may use two types of storage. In many implementations, a tailof a Stream may be in so-called Tier-1 storage. In most implementations,writes may be implemented by Apache BookKeeper as described herein. Incertain implementations, tail reads may be served out of a managedmemory cache. In fact, in some implementations BookKeeper may servereads in failure recovery scenarios, for example, when a crash hasoccurred and the system is being recovered. In some implementations, thehistorical part of the Stream may be in Tier-2 storage that may beoptimized for low cost storage with high throughput. In someimplementations, Pravega may use efficient in-memory read ahead cache,taking advantage of the fact that Streams are usually read in largecontiguous chunks and that HDFS is well suited for those sort of large,high throughput reads. In most implementations, tail reads may notimpact the performance of writes.

Infinite Retention

In many implementations, data in Streams may be retained indefinitely.In most implementations, the current disclosure enables an API to accessboth real-time and historical data. In certain implementations, batchand real-time applications may be handled efficiently.

In some implementations, there may be a value to keeping old data. Insome implementations, for example in machine learning, it may bedesirable to periodically change the model and train the new version ofthe model against as much historical data as possible to yield a moreaccurate predictive power of the model. In many implementations, thecurrent disclosure has auto-tiering and enables keeping historical datathat does not affect the performance of tail reads and writes.

In many implementations, the size of a stream may not be limited by thestorage capacity of a single server, but rather, may be limited by thestorage capacity of a data center or a cloud provider. In furtherimplementations, as cost of storage decreases, the economic incentive todelete data may go away.

Storage Efficiency

In many implementations, the current disclosure enables buildingpipelines of data processing, combining batch, real-time and otherapplications without duplicating data for every step of the pipeline.

FIG. 4 illustrates an example data processing environment for processingstreaming application data in a conventional system. Consider a dataprocessing environment as depicted that combines real time processingusing Spark, Flink, and or Storm; Hadoop Distributed File System(“HDFS”) for batch processing; some kind of Lucene-based Searchmechanism like Elastic Search for full text search; and maybe one (orseveral) NoSQL databases to support micro-services apps.

Using traditional approaches, one set of source data, for example sensordata from an IoT app, would be ingested and replicated separately byeach system. Conventionally there would be with 3 replica copies of thedata protected in the pub/sub middleware system, 3 copies of the data inHDFS, 3 copies of the data in Lucene, 3 copies of the data in the NoSQLdatabase, and potential 3 copies of the data in other support servicesdesirable to the system. When we consider the source data is measured interabytes, the cost of data replication separated by middleware categorybecomes prohibitively expensive.

FIG. 5 illustrates an example data processing environment for processingstreaming application data in accordance with implementations of thisdisclosure. In this implementation, consider the same pipeline from FIG.4 using techniques of the current disclosure and middleware adapted touse techniques of the current disclosure for its storage.

In implementations of the current disclosure, there may be a singlesource of truth for data processing. In many implementations, with tier2 storage, and features like erasure coding, 1 TB of data may beprotected using less than 2 TB of storage. In further implementations,data may be ingested and protected in one place. Implementations of thecurrent disclosure enable one place for storage and may provide a singlesource of truth for the entire pipeline. In further implementations, ifthe bulk of the data being stored in Tier-2 locations that are enabledwith erasure coding to efficiently protect the data, the storage cost ofthe data may be substantially reduced.

Durability

In some implementations, there may not be compromise betweenperformance, durability and consistency. In many implementations, thecurrent disclosure may enable durable storage of streaming data withstrong consistency, ordering guarantees and great performance. In mostimplementations, durability may be a fundamental storage primitiverequirement. Conventionally, storage that could lose data is notreliable storage. Usually, systems based on storage can lose data maynot be production quality.

In some implementations, once a write operation is acknowledged, data ofthe write may not be lost, even when failures occur. In manyimplementations, data may not be lost as data may be saved in protected,persistent storage before the write operation returns to the Writer. Inmost implementations, data in a Stream may be protected. In manyimplementations, a Stream may be treated as a system of record, similarto databases or files.

Transaction Support

In many implementations, the current disclosure may ensure that a set ofevents are written to a stream atomically.

In some implementations, a Pravega Transaction may be part of aPravega's writer API. In certain implementations, data may be written toa Stream through the API, or an application may write data through aTransaction. In some implementations, with Transactions, a Writer maypersist data at a one point in time, and later decide whether the datashould be appended to a Stream or abandoned.

In many implementations, using a Transaction, data may be written to aStream when the Transaction is committed. In most implementations, whena Transaction is committed, data written to the Transaction may beatomically appended to the Stream. In many implementations, asTransactions may be implemented in the same way as Stream Segments, datawritten to a Transaction is just as durable as data written directly toa Stream. In certain implementations, if a Transaction is abandoned(e.g. if the writer crashes) the Transaction may be aborted and all datamay be discarded. In some implementations, an application may choose toabort the Transaction through the API if a condition occurs thatsuggests the writer should discard the data.

In many implementations, transactions may enable chaining Flink jobstogether. In some implementations, when a Flink job uses Pravega as asink, it may begin a Transaction, and if it successfully finishesprocessing, it may commit the Transaction, writing the data into a basedsink. In some implementations, if the job fails a Transaction may timeout and data may not be written. In some implementations, when a job isrestarted, there may be no “partial result” in a sink that may need tobe managed or cleaned up.

In some implementations, by combining Transactions, it may be possibleto chain Flink jobs together, having one job's Pravega-based sink be thesource for a downstream Flink job. In some implementations, the currentdisclosure may provide the ability for a pipeline of Flink jobs to haveend-end exactly once, guaranteed ordering of data processing.

In some implementations, it may be possible for Transactions acrossmultiple Streams to be coordinated with Transactions, so that a Flinkjob may use 2 or more Pravega-based sinks to provide source input todownstream Flink jobs. In additional implementations, it may be possiblefor application logic to coordinate Pravega Transactions with externaldatabases such as Flink's checkpoint store.

Example Distributed System

FIG. 6 illustrates an example Pravega Cluster in accordance withimplementations of this disclosure. Implementations of the currentdisclosure may be deployed as a distributed system. In a particularimplementation, there may be a cluster of servers and storagecoordinated to run Pravega called a “Pravega cluster.” In someimplementations, implementations of the current disclosure may becomposed of two kinds of workload: Controller instances and PravegaNodes.

In some implementations, a set of Controller instances may make up acontrol plane, which may provide functionality to create, alter anddelete Streams, retrieve information about Streams, monitor the healthof a Pravega cluster, gather metrics etc. In most implementations, theremay be multiple (recommended at least 3) Controller instances running ina cluster for high availability.

In some implementations, a set of Pravega Nodes in a cluster maycomprise a data plane or Pravega data plane. In many implementationsservers, or Pravega Nodes, may provide an API to read and write data inStreams. In certain implementations, data storage may be comprised oftwo tiers: Tier 1 Storage, which provides short term, low-latency, datastorage, guaranteeing the durability of data written to Streams and Tier2 Storage providing longer term storage of Stream data. In certainimplementations, Apache Bookkeeper may be used to implement Tier 1Storage and can use HDFS, DellEMC's Isilon OneFS or DellEMC's ElasticCloud Storage (ECS) to implement Tier 2 Storage. In someimplementations, Tier 1 Storage may run within a Pravega cluster. Inmost implementations, Tier 2 Storage may be deployed outside the Pravegacluster.

In many implementations, tiering storage may be important to deliver thecombination of fast access to Stream data but also allow Streams tostore a vast amount of data. In some implementations, Tier 1 storage maypersist the most recently written Stream data. In some implementations,as data in Tier 1 Storage ages, it may be moved into Tier 2 Storage.

Implementations of the current disclosure may use Apache Zookeeper as acoordination mechanism for the components in the Pravega cluster. Inother implementations, other components, including Pravega componentssuch as BookKeeper may rely on Zookeeper.

In some implementations, the current disclosure may provide a clientlibrary, written in Java, for building client-side applications such asanalytics applications using Flink. In some implementations herein, aclient library may be referred to as a Pravega client library. Incertain implementations, a Pravega Java Client Library may manage theinteraction between application code and Pravega via a custom TCP wireprotocol.

Streams

In certain implementations, the current disclosure may organize datainto Streams. In most implementations, a stream may be a persistent,unbounded, append-only collection of data records called Events. In someimplementations herein, a stream may be referred to as a Pravega Stream.

In certain implementations, a Pravega Stream may be based on anappend-only log data structure. In some implementations, by usingappend-only logs, the current implementation enable rapid ingestion ofinformation into durable storage, and support a large variety ofapplication use cases such as publish/subscribe messaging similar toApache Kafka, NoSQL databases such as a Time Series Database (TSDB),workflow engines, event-oriented applications and many other kinds ofapplications.

In some implementations, when a Stream is created, the Stream may begiven a meaningful name such as “IoTSensorData” or“WebApplicationLog_201 7-03-30T13:08:10.” In certain implementations, aStream's name may help other developers understand the kind of data thatis stored in the Stream. In most implementations, a Pravega Stream namesmay be organized within a Scope. In most implementations, a Scope may bea string and may convey some sort of meaning to developers such as“FactoryMachines” or “HR_Website_logs.” In some implementations, a Scopemay act as a namespace for Stream names. Stream names can be uniquewithin a Scope.

In many implementations, a Stream may be uniquely identified by acombination of its Stream Name and Scope. In certain implementations,Scope may be used to segregate names by tenant (in a multi tenantenvironment), by department of an organization, by geographic locationor any other categorization the developer chooses.

In some implementations, a Stream may be unbounded in size. In manyimplementations, there may not be a limit on how many Events can be in aStream or how many total bytes are stored in a Stream. In mostimplementations, Pravega may be a data storage primitive. In certainimplementations, the amount of data stored in Pravega may be limited bythe total storage capacity of the data center. In most implementations,once data is written to it, it may be durably stored.

In some implementations, to deal with a potentially large amount of datawithin a Stream, Pravega Streams may be divided into Stream Segments. Incertain implementations, a Stream Segment may be a shard, or partitionof the data within a Stream.

Events

In many implementations, a primitive data element in Pravega may be anEvent. In some implementations, an Event may be a collection of byteswithin a Stream. In certain implementations, an Event may be as simpleas a small number of bytes containing a temperature reading from an IoTsensor composed of a timestamp, a metric identifier and a value. In someimplementations, an event may be web data associated with a user clickon a website. In certain implementations, an event may be anything youcan represent as a collection of bytes. In many implementations,applications may make sense of Events using standard Java serializersand deserializers, allowing them to read and write objects in Pravegausing similar techniques to reading and writing objects from file-basedstorage.

In most implementations, an Event may have a Routing Key. In certainimplementations, a Routing Key may be a String that allows Pravega andapplication developers to determine which Events are related. In furtherimplementations, a Routing Key may be a string that developers use togroup similar Events together. In certain implementations, a Routing Keymay be derived from data naturally occurring in an Event, something like“customer-id” or “machine-id”, but it may also be some artificial Stringsuch as a UUID or a monotonically increasing number. In manyimplementations, a Routing Key may be something like a timestamp (togroup Events together by time) or perhaps a Routing Key could be an IoTsensor id (to group Events by machine). In certain implementations, aRouting Key may be important to defining precise read and writesemantics.

Readers, Writers and ReaderGroups

FIG. 7 illustrates an example Pravega Cluster ingesting data from a setof writers and making data available to a set of reader groups inaccordance with implementations of this disclosure. In someimplementations, the current disclosure provides for a client library,written in Java that may implement a convenient API for Writer andReader applications to use. In some implementations, a Pravega JavaClient Library may encapsulate a wire protocol used to communicatebetween Pravega clients and Pravega.

In some implementations, a Writer may be an application that createsEvents and writes them into a Stream. In most implementations, data maybe written by appending to the tail (front) of a Stream.

In certain implementations, a Reader may be an application that readsEvents from a Stream. In many implementations, a Reader may read fromany point in the Stream. In certain implementations, a Reader may bereading Events from the tail of the Stream. In certain implementations,Events may be delivered to Readers as quickly as possible. For example,in one implementation, Events can be delivered to Readers within tens ofmilliseconds after they were written. In further implementations,Readers may read from earlier parts of the Stream (called catch-upreads). In certain implementations, a Position may represent point in aStream a Reader is currently located. In some implementations, aPosition object may be used as a recovery mechanism—applications thatpersist the last Position a Reader has successfully processed can usethat information to initialize a replacement Reader to pickup where afailed Reader left off. In certain implementations, using this patternof persisting Position objects, an application may be built thatguarantees exactly once Event processing in the face of Reader failure.

In some implementations, Readers may be organized into ReaderGroups. Inmany implementations, a ReaderGroup may be a named collection of Readersthat together, in parallel, read Events from a given Stream. In certainimplementations, when a Reader is created through the Pravega data planeAPI, the name of the ReaderGroup may be part of it. In certainimplementations, each Event published to a Stream may be guaranteed tobe sent to one Reader within the ReaderGroup. In many implementations,there may be one Reader in the ReaderGroup or there may be many. Infurther implementations, there may be many different ReaderGroupssimultaneously reading from any given Stream.

In certain implementations, ReaderGroup may be a “composite Reader” or a“distributed Reader”, that allows a distributed application to read andprocess Stream data in parallel, such that a massive amount of Streamdata can be consumed by a coordinated fleet of Readers in a ReaderGroup.In some implementations, a collection of Flink tasks processing Streamdata in parallel may be an example use of a ReaderGroup.

Stream Segments

FIG. 8 illustrates an example Stream decomposed into a set of StreamSegments in accordance with implementations of this disclosure. In manyimplementations, a Stream may be decomposed into a set of StreamSegments. A Stream Segment is a shard or partition of a Stream. In manyimplementations, an Event may be Stored within a Stream Segment

In some implementations, a Stream Segment may be a container for Eventswithin the Stream. In certain implementations, when an Event is writteninto a Stream, it may be stored in one of the Stream Segments based onthe Event's Routing Key. In certain implementations, consistent hashingmay be used to assign Events to Stream Segments. In certainimplementations, event Routing Keys may be hashed to form a “key space.”In some implementations, key space may be divided into a number ofpartitions, corresponding to the number of Stream Segments. In someimplementations, consistent hashing may determine which Segment an Eventmay be assigned to.

Auto Scaling Stream Segments

In some implementations. AutoScaling may refer to a number of StreamSegments that vary over time. In some implementations, a number ofStream Segments in a Stream may grow and shrink over time as I/O load onthe Stream increases and decreases, which may be referred to herein asAutoScaling.

FIG. 9 illustrates an example graph depicting the relationship betweenrouting keys, stream segments, and time in accordance withimplementations of this disclosure. In this example implementation, aStream starts out at time t₀ with a configurable number of Segments. Ifthe rate of data written to the Stream is constant, the number ofSegments won't change.

However at time t1, the system noted an increase in the ingestion rateand chose to split Segment 1 into two parts. We call this a Scale-upEvent. Before t1, Events with a Routing Key that hashes to the upperpart of the key space (values 200-399) would be placed in Segment 1 andthose that hash into the lower part of the key space (values 0-199)would be placed in Segment 0. After t1, Segment 1 is split into Segment2 and Segment 3. Segment 1 is sealed and it no longer accepts writes. Atthis point in time, Events with Routing Key 300 and above are written toSegment 3 and those between 200 and 299 would be written into Segment 2.Segment 0 still keeps accepting the same range of Events as before t1.

In this implementation, there is another Scale-up Event at time t₂ asSegment 0's range of Routing Key is split into Segment 5 & Segment 4.Also at this time, Segment 0 is sealed off so that it accepts no furtherwrites. Segments covering a contiguous range of the key space can alsobe merged. At time t3, Segment 2's range and Segment 5's range aremerged into Segment 6 to accommodate a decrease in load on the Stream.

In some implementations, when a Stream is created, it may be configuredwith a Scaling Policy that determines how a Stream reacts to changes inits load.

In certain implementation, there may be one or more of the followingScaling Policies. (1) Fixed—The number of Stream Segments may not varywith load. (2) Size-based—As the number of bytes of data per secondwritten to the Stream increases past a certain target rate, the numberof Stream Segments may be increased. If it falls below a certain level,the number of Stream Segments may be decreased. (3) Event-based—Similarto Size-based Scaling Policy, except it may use the number of Eventsinstead of the number of bytes.

Events, Stream Segments and Autoscaling

FIG. 10 illustrates an example graph depicting Events and theirrelationship between routing keys, stream segments, and time inaccordance with implementations of this disclosure. In someimplementations, an Event may be written into one of the Stream'sSegments. In some implementations, Stream Segments may be a bucketing ofEvents based on Routing Key and time. In many implementations, Eventspublished to a Stream within a given value of Routing Key may appear inthe same Stream Segment.

In some implementations, Events may be written into the active StreamSegments. In many implementations, Segments that are sealed may notaccept writes. In FIG. 10, at time “now”, Stream Segments 3, 6 and 4 areactive and between those three Stream Segments the entire key space iscovered.

Stream Segments and Reader Groups

FIG. 11 illustrates an example of Stream Segments and associatedReaderGroups in accordance with implementations of this disclosure. Incertain implementations, a Reader in a ReaderGroup may be assigned zeroor more Stream Segments from which to read. In many implementations, thenumber of Stream Segments may be balanced to which Reader it isassigned. In FIG. 11, Reader B1 reads from 2 Stream Segments while eachof the other Readers in the Reader Group have only 1 Stream Segment toread from. In certain implementations, each Stream Segment may be readby one Reader in any ReaderGroup configured to read from that Stream. Inmany implementations, Readers may be added to the ReaderGroup, orReaders crash and may be removed from the ReaderGroup. In manyimplementations Stream Segments so that Stream Segments may bereassigned to be balanced amongst the Readers.

In some implementations, a number of Stream Segments in a Stream maydetermine the upper bound of parallelism of readers within aReaderGroup. In many implementations, the more Stream Segments, the moreseparate, parallel sets of Readers may consume a Stream. In FIG. 11,Streaml has 4 Stream Segments. That means that the largest effectiveReader Group would contain 4 Readers. Reader Group named “B” in theabove figure is not quite optimal. If one more Reader was added to theReaderGroup, each Reader would have 1 Stream Segment to process,maximizing read parallelism. However, the number of Readers in theReaderGroup increases beyond 4, at least one of the Readers will not beassigned a Stream Segment. If Stream 1 in the example implementationabove experienced a Scale-Down Event, reducing the number of StreamSegments to 3, then Reader Group B as depicted would have an idealnumber of Readers.

In some implementations, an Auto Scaling feature may enable dynamicallydetermining the right size of a Stream. In most implementations, PravegaStreams may grow and shrink to match the behavior of the data input. Inmost implementations, a size of any Stream may be limited by the totalstorage capacity made available to the Pravega cluster. In manyimplementations, if bigger streams are needed, more storage may be addedto a cluster.

In certain implementations, applications may be made aware of changes ina Stream's Segments. In some implementations, the current disclosure mayprovide hooks to allow applications to react to changes in the number ofSegments in a Stream, adjusting the number of Readers within aReaderGroup, to maintain optimal read parallelism if resources allow. Incertain implementations, this may enable a Flink application to increaseor decrease the number of task instances that are processing a Stream inparallel, as scale events occur over time.

In certain implementations, the current disclosure may enable: (1)Events with the same Routing Key are consumed in the order they werewritten; (2) Events with different Routing Keys sent to a specificReader will always be seen in the same order even if the Reader backs upand re-reads them; and (3) If there are multiple Readers reading aStream and they all backup and replay the Stream, they will each see thesame Events in the same order each time.

ReaderGroup Checkpoints

In certain implementations, the current disclosure may enable ability anapplication to initiate a Checkpoint on a ReaderGroup. In manyimplementations, a Checkpoint may create a consistent “point in time”persistence of the state of each Reader in the ReaderGroup, by using aspecialized Event (a Checkpoint Event) to signal each Reader to preserveits state. In certain implementations, once a Checkpoint has beencompleted, the application may use the Checkpoint to reset all theReaders in the ReaderGroup to the known consistent state represented bythe Checkpoint.

Transactions

FIG. 12 illustrates an example graph depicting Events published into aTransaction and their relationship between routing keys, streamsegments, and time in accordance with implementations of thisdisclosure. In certain implementation, Pravega may support Transactions.In some implementation, a Transaction may “batch” up a bunch of Eventsand commit them as a unit into a Stream. In certain implementations, aFlink job may continuously produce results of some data processing anduse the Transaction to durably accumulate the results of the processing.In certain implementations, at the end of a time window (for example)the Flink job may commit the Transaction and therefore may make theresults of the processing available for downstream processing, or in thecase of an error, abort the Transaction and the results disappear.

In certain implementations, a difference between Pravega's Transactionsand similar approaches (such as Kafka's producer-side batching) isrelated to durability. Events added to a Transaction are durable whenthe Event is ack'd back to the Writer. However, the Events in theTransaction are not visible to readers until the Transaction iscommitted by the Writer. A Transaction is a lot like a Stream; aTransaction is associated with multiple Stream Segments. When an Eventis published into a Transaction, the Event itself is appended to aStream Segment of the Transaction. Say a Stream had 5 Segments, when aTransaction is created on that Stream, conceptually that Transactionalso has 5 Segments. When an Event is published into the Transaction, itis routed to the same numbered Segment as if it were published to theStream itself (if the Event would have been placed in Segment 3 in the“real” Stream, then it will appear in Segment 3 of the Transaction).When the Transaction is committed, each of the Transaction's Segments isautomatically appended to the corresponding Segment in the real Stream.If the Stream is aborted, the Transaction, all its Segments and all theEvents published into the Transaction are removed from Pravega.

In certain implementations, events published into a Transaction may notbe visible to the Reader until that transaction is committed.

State Synchronizers

FIG. 13 illustrates an example state synchronizer in accordance withimplementations of this disclosure. In many implementations, Pravega maybe a streaming storage primitive. In certain implementations, thecurrent disclosure may provide a mechanism to coordinate processes in adistributed computing environment. In certain implementations, a StateSynchronizer feature of Pravega may fall into the latter category.

In certain implementation, a State Synchronizer may use a Pravega Streamto provide a synchronization mechanism for state shared between multipleprocesses running in a cluster, making it easier to build distributedapplications. In most implementations, with a State Synchronizer, thecurrent disclosure may enable reads and changes to be made to sharedstate with consistency and optimistic locking.

In certain implementations, a State Synchronizer may be used to maintaina single, shared copy of an application's configuration property acrossall instances of that application in a cloud. In some implementations, aState Synchronizer may be used to store one piece of data or a map withthousands of different key value pairs. In some implementations, a StateSynchronizer may be used to manage a state of ReaderGroups and Readersdistributed throughout the network.

In some implementation, a State Synchronizer may be created on a Streamin a fashion similar to how an EventWriter is created. In manyimplementations, a State Synchronizer may keep a local copy of theshared state to make access to the data really fast for the application.In certain implementations, changes to shared state may be writtenthrough the State Synchronizer to the Stream keeping track of allchanges to the shared state. In some implementations, each applicationinstance may use a State Synchronizer to stay up to date with changes bypulling updates to shared state and modifying the local copy of thedata. In many implementations, consistency may be maintained through aconditional append style of updates to the shared state through theState Synchronizer, making sure that updates are made only to the mostrecent version of the shared state.

In many implementations, a State Synchronizer may occasionally be“compacted”, compressing and removing old state updates so that only themost recent version of the state is kept in the backing stream. Incertain implementations, a compacting feature may help app developersmake sure that shared state does not grow unchecked.

In some implementations, a State Synchronizer may work best when mostupdates to shared state are small in comparison to the total data sizebeing stored, allowing them to be written as small deltas. In manyimplementations, as with any optimistic concurrency system, a StateSynchronizer may not be at its best when many processes are allattempting to simultaneously update the same piece of data.

Example Component Level Implementation

FIG. 14 illustrates an example implementation of a set of streamingclients, a Pravega Cluster, the autoscaling of segments, and tieredstorage in accordance with implementations of this disclosure.

Pravega clients are Writers and Readers. Writers write Events into aStream. Readers read Events from a Stream. Readers are grouped intoReaderGroups to read from a Stream in parallel.

The Controller is a server-side component that manages the control planeof Pravega. Streams are created, updated and listed using the ControllerAPI.

The Pravega Server or Pravega Node is a server-side component thatimplements read, write and other data plane operations.

Streams are the fundamental storage primitive in Pravega. Streamscontain a set of data elements called Events. Events are appended to the“tail” of the Stream by Writers. Readers can read Events from anywherein the Stream.

A Stream is partitioned into a set of Stream Segments. The number ofStream Segments in a Stream can change over time. Events are writteninto exactly one of the Stream Segments based on Routing Key. For anyReaderGroup reading a Stream, each Stream Segment is assigned to oneReader in that ReaderGroup.

Each Stream Segment is stored in a combination of Tier 1 and Tier 2storage. The tail of the Segment is stored in Tier 1 providing lowlatency reads and writes. The rest of the Segment is stored in Tier2,providing high throughput read access with near-infinite scale and lowcost.

In many implementations, to deliver an efficient implementation ofStreams, the current disclosure may include a tiered storage model. Inmany implementations, events may be persisted in low latency/high TOPSstorage (Tier 1 Storage) and higher throughput storage (Tier 2 Storage).In most implementations, Writers and Readers may be oblivious to thetiered storage model from an API perspective.

Append Only Log

FIG. 15 illustrates an example append-only log in accordance withimplementations of this disclosure. In some embodiments, write activityand much of the read activity may happen at an end tail of the log. Inmany embodiments, writes may be appended to a log and many this mayenable reads to the data as fast as it arrives in the log. In manyembodiments, data access mechanisms described herein may be dominated bythe need for low-latency—low latency writes by Writers and near realtime access to the published data by Readers.

In certain embodiments, not all Readers may read from the tail of thelog. In some embodiments, certain Readers may want to read starting atsome arbitrary position in a log. In some embodiments, reads from anarbitrary position may be referred to as catch-up reads. Conventionally,access to historical data traditionally was done by batch analyticsjobs, often using HDFS and Map/Reduce. In many embodiments, usingtechniques of the instant disclosure, it may be possible to accesshistorical data as well as current data by just accessing the log. In aparticular embodiment, the historical data may be stored in SSDs. Insome embodiments, the current disclosure may provide a mechanism thatallows customers to use cost-effective, highly-scalable, high-throughputstorage for the historical part of the log, to enable retention ofhistorical data.

In some embodiments, Tier 1 Storage may be used to make writing toStreams fast and durable and to make sure reading from the tail of aStream is as fast as possible. In certain embodiments, Tier 1 Storagemay be implemented on faster SSDs or even non-volatile RAM.

In many embodiments, Tier 2 Storage may provide a highly-scalable,high-throughput cost-effective storage. In some embodiments, Tier 2 maybe deployed on spinning disks. In certain embodiments, the currentdisclosure may enable asynchronous migration of Events from Tier 1 toTier 2 to reflect the different access patterns to Stream data. Incertain embodiments, Tier 2 Storage may be based on an HDFS model.

FIGS. 16-18 illustrate methods and/or flow diagrams in accordance withthis disclosure. For simplicity of explanation, the methods are depictedand described as a series of acts. However, acts in accordance with thisdisclosure can occur in various orders and/or concurrently, and withother acts not presented and described herein. Furthermore, not allillustrated acts may be required to implement the methods in accordancewith the disclosed subject matter. In addition, those skilled in the artwill understand and appreciate that the methods could alternatively berepresented as a series of interrelated states via a state diagram orevents. Additionally, it should be appreciated that the methodsdisclosed in this specification are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethods to computing devices. The term article of manufacture, as usedherein, is intended to encompass a computer program accessible from anycomputer-readable device or storage media.

Moreover, various acts have been described in detail above in connectionwith respective system diagrams. It is to be appreciated that thedetailed description of such acts in the prior figures can be and areintended to be implementable in accordance with one or more of thefollowing methods.

Referring now to FIG. 16, there is illustrated an example flow diagrammethod for organizing present and future reads from a tiered streamingdata storage layer in accordance with implementations of thisdisclosure.

At 1602, a lookup request can be received from a caller, wherein thelookup request includes a segment, an offset, and a maximum length.

At 1610, an iterator can be established based on the segment and theoffset, that returns one of outcomes described in steps 1612-1616 tocaller for each iteration until the maximum length.

At 1612, a set of data responsive to the lookup request can be sent tothe caller, wherein the next set of data is stored in local cachememory. It can be appreciated that local cache memory and tier 1 storageare synonymous.

At 1614, a storage read result can be sent to the caller.

At 1616, a future read result can be sent to the caller.

Referring now to FIG. 17, there is illustrated an example flow diagrammethod for organizing present and future reads from a tiered streamingdata storage layer including processing storage read results inaccordance with implementations of this disclosure. Steps 1702-1716 canproceed as describe above with respect to steps 1602-1616 on FIG. 16.

At 1720, a notification can be received from the caller that the callerhas accepted the storage read result.

At 1722, a set of data responsive to the storage read can be read fromtier 2 storage. In one implementation, additional sequential data canalso be read from tier 2 storage in anticipation of future iterations bythe caller or future read results by the caller, reducing the need forsubsequent storage read results for contiguous data.

At 1724, data read from tier 2 can be stored in local cache memory.

At 1726, the set of data can be returned to the caller.

Referring now to FIG. 18, there is illustrated an example flow diagrammethod for organizing present and future reads from a tiered streamingdata storage layer including processing future read results inaccordance with implementations of this disclosure. Steps 1802-1816 canproceed as described above with respect to steps 1602-1616 on FIG. 16.

At 1820, a notification can be received from the caller that the callerhas accepted the future read result.

At 1822, the future read can be registered in a read index.

At 1824, the iterator can be paused in response to registering thefuture read in the read index.

At 1826, in response to the read index indicating that a segmentassociated with the future read has a new append, a set of dataresponsive to the future read can be determined, wherein determining theset of data responsive to the future read can restart the iterator. Inone implementation, when an append occurs on a segment, that data isinserted into the cache and associated with the segment name and offset.The index is updated to reflect the append, and any future reads thatregistered to the location are triggered.

At 1828, the set of data can be returned to the caller.

The illustrated aspects of the disclosure can be practiced indistributed computing environments where certain tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules can belocated in both local and remote memory storage devices.

The systems and processes described above can be embodied withinhardware, such as a single integrated circuit (IC) chip, multiple ICs,an application specific integrated circuit (ASIC), or the like. Further,the order in which some or all of the process blocks appear in eachprocess should not be deemed limiting. Rather, it should be understoodthat some of the process blocks can be executed in a variety of ordersthat are not all of which may be explicitly illustrated herein.

What has been described above includes examples of the implementationsof the present disclosure. It is, of course, not possible to describeevery conceivable combination of components or methods for purposes ofdescribing the claimed subject matter, but many further combinations andpermutations of the subject innovation are possible. Accordingly, theclaimed subject matter is intended to embrace all such alterations,modifications, and variations that fall within the spirit and scope ofthe appended claims. Moreover, the above description of illustratedimplementations of this disclosure, including what is described in theAbstract, is not intended to be exhaustive or to limit the disclosedimplementations to the precise forms disclosed. While specificimplementations and examples are described herein for illustrativepurposes, various modifications are possible that are considered withinthe scope of such implementations and examples, as those skilled in therelevant art can recognize.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms used to describe such components are intended to correspond,unless otherwise indicated, to any component which performs thespecified function of the described component (e.g., a functionalequivalent), even though not structurally equivalent to the disclosedstructure, which performs the function in the herein illustratedexemplary aspects of the claimed subject matter. In this regard, it willalso be recognized that the innovation includes a system as well as acomputer-readable storage medium having computer-executable instructionsfor performing the acts and/or events of the various methods of theclaimed subject matter.

What is claimed is:
 1. A method comprising: receiving a lookup requestfrom a caller, wherein the lookup request includes a segment, an offset,and a maximum length; and establishing an iterator based on the segmentand the offset that returns one of the following to the caller for eachiteration until the maximum length: a next set of data responsive to thelookup request wherein the next set of data is stored in local cachememory; a storage read result; or a future read result.
 2. The method ofclaim 1, further comprising: receiving a notification from the callerfor a storage read based on the storage read result; reading a set ofdata responsive to the storage read from tier 2 storage; storing the setof data in local cache memory; and returning the set of data to thecaller.
 3. The method of claim 1, further comprising: receiving anotification from the caller for a future read based on the future readresult; registering the future read in a read index; in response to theread index indicating that a segment associated with the future read hasa new append, determining a set of data responsive to the future read;and returning the set of data to the caller.
 4. The method of claim 3,further comprising: pausing the iterator in response to registering thefuture read in the read index, wherein determining the set of dataresponsive to the future read restarts the iterator.
 5. The method ofclaim 1, wherein the segment identifier and the offset are a key and acache location is a value in a key value database.
 6. The method ofclaim 1, wherein the iterator returns the future read result based on acurrent read offset reaching an end of the segment.
 7. A systemcomprising at least one storage device and at least one hardwareprocessor configured to: receive a lookup request from a caller, whereinthe lookup request includes a segment, an offset, and a maximum length;and establish an iterator based on the segment and the offset thatreturns one of the following to the caller for each iteration until themaximum length: a next set of data responsive to the lookup requestwherein the next set of data is stored in local cache memory; a storageread result; or a future read result.
 8. The system of claim 7, furtherconfigured to: receive a notification from the caller for a storage readbased on the storage read result; read a set of data responsive to thestorage read from tier 2 storage; store the set of data in local cachememory; and return the set of data to the caller.
 9. The system of claim7, further configured to: receiving a notification from the caller for afuture read based on the future read result; registering the future readin a read index; in response to the read index indicating that a segmentassociated with the future read has a new append, determining a set ofdata responsive to the future read; and returning the set of data to thecaller.
 10. The system of claim 9, further configured to: pausing theiterator in response to registering the future read in the read index,wherein determining the set of data responsive to the future readrestarts the iterator.
 11. The system of claim 7, wherein the segmentidentifier and the offset are a key and a cache location is a value in akey value database.
 12. The system of claim 7, wherein the iteratorreturns the future read result based on a current read offset reachingan end of the segment.
 13. A non-transitory computer readable mediumwith program instructions stored thereon to perform the following acts:receiving a lookup request from a caller, wherein the lookup requestincludes a segment, an offset, and a maximum length; and establishing aniterator based on the segment and the offset that returns one of thefollowing to the caller for each iteration until the maximum length: anext set of data responsive to the lookup request wherein the next setof data is stored in local cache memory; a storage read result; or afuture read result.
 14. The non-transitory computer readable medium ofclaim 13, with program instructions stored thereon to further performthe following acts: receiving a notification from the caller for astorage read based on the storage read result; reading a set of dataresponsive to the storage read from tier 2 storage; storing the set ofdata in local cache memory; and returning the set of data to the caller.15. The non-transitory computer readable medium of claim 13, withprogram instructions stored thereon to further perform the followingacts: receiving a notification from the caller for a future read basedon the future read result; registering the future read in a read index;in response to the read index indicating that a segment associated withthe future read has a new append, determining a set of data responsiveto the future read; and returning the set of data to the caller.
 16. Thenon-transitory computer readable medium of claim 15, with programinstructions stored thereon to further perform the following acts:pausing the iterator in response to registering the future read in theread index, wherein determining the set of data responsive to the futureread restarts the iterator.
 17. The non-transitory computer readablemedium of claim 13, wherein the segment identifier and the offset are akey and a cache location is a value in a key value database.
 18. Thenon-transitory computer readable medium of claim 13, wherein theiterator returns the future read result based on a current read offsetreaching an end of the segment.